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Executive Summary 


Introduction 


With the support of the Noyce Foundation, National 4-H Council has contracted with 
Policy Studies Associates (PSA) to evaluate the implementation of the 4-H Science, Engineering 
and Technology (SET) Initiative. The goals of this initiative are to increase the number and 
quality of science, engineering, and technology programs that 4-H offers around the country, and 
to increase the number of youth involved in these programs. Objectives for the SET initiative 
are to increase SET interest and literacy among youth, to increase the number of youth pursuing 
post-secondary education in SET, and to increase the number of youth pursuing SET careers. 


Annually, nearly six million youth participate in 4-H, which is implemented by 106 
Land-Grant Universities and Colleges (LGUs) in more than 3,000 counties as a part of the 
Cooperative Extension System. National leadership is provided by 4-H National Headquarters at 
the National Institute for Food and Agriculture, USDA, and National 4-H Council, which is the 
national nonprofit partner of 4-H and the Cooperative Extension System. National 4-H Council 
focuses on fundraising, branding, communications, and legal and fiduciary support to 4-H 
programs. 


4-H has set an enrollment goal: by the end of 2013, one million youth who have never 
before been in 4-H will enroll in SET programs. In order to meet this goal, 4-H has sought since 
2006 to increase its capacity and infrastructure for providing 4-H SET programming. As part of 
this effort, 4-H formed the National 4-H SET Leadership Team, which consists of national, state 
and county-level 4-H professionals. In addition, SET Liaisons have been appointed at Land 
Grant Universities (LGUs) around the country to help implement SET programs and to recruit 
youth into these programs. 


The evaluation’s winter 2009 report on the implementation of SET programs nationwide 
suggested that SET programs offer programming on a variety of SET-related topics through a 
range of delivery methods. Some SET programs deliver content on topics that have traditionally 
been covered by 4-H programming, while other SET programs offer youth the opportunity to 
explore topics that are new to 4-H, such as rocketry and robotics. The National 4-H SET 
Leadership Team created an outline of SET program requirements, called the 4-H SET 
Checklist, for universities to use as a guide in the development of SET programs.' Generally, 
this Checklist sets out the expectation that SET programs: be inquiry-based and grounded in 
experiential learning; involve building SET-related skills; and, like all 4-H programs, employ 
positive youth development practices. 


In addition to the 4-H SET Checklist, 4-H has also developed a logic model for the SET 
initiative. According to the 4-H SET Logic Model, programs should be designed with the 
following short-term youth outcomes in mind: 


' The 4-H SET Checklist is reproduced in Appendix D. 


- Increased awareness of SET 


a Improved SET skills (scientific methods) and knowledge (content areas) 
a Increased awareness of opportunities to contribute to society using SET skills 
7 Increased life skills (self-efficacy) 

Methods 


Youth survey design. As part of the evaluation of SET programs, PSA worked with the 
4-H SET Instrument Design Team: Melissa Cater, Mary Arnold, Lisa Bouillion Diaz, Katherine 
Heck, June Mead, Beverly Spears, Ben Silliman, and Maureen Mulroy, and with Jill Walahoski 
and Suzanne LeMenestrel to develop the 4-H SET Youth Engagement, Attitudes and Knowledge 
(YEAK) survey for administration to SET participants. In this first year of the evaluation, the 
YEAK survey serves to address the following questions about the youth in the survey sample: 


C7] What are the characteristics of the youth involved in SET programming? 


7 What are participants’ attitudes toward the SET fields? Do participants aspire to 
pursue career opportunities in the SET fields? 


7 What level of education do participants want to achieve? 


7 To what extent are participants engaged in both formal and informal SET 
learning? Do participants pursue SET leadership opportunities? 


2 What SET-related skills, abilities, and knowledge do participants have? 
7 What factors, if any, are associated with participants’ engagement, attitudes, and 
knowledge of SET? 


Sample selection. In order to create a sampling frame from which to draw a sample of 
youth, PSA surveyed county- and state-level 4-H staff and asked them to identify 4-H SET 
programs where youth could be surveyed. From the list of SET programs generated, PSA drew a 
random sample of SET programs for the youth survey. In drawing this sample, PSA ensured that 
programs: (1) represented all five regions, (2) were located in communities of all sizes, (3) 
enrolled participants of varying ages, and (4) utilized a variety of delivery methods. 


Response rate. PSA sent paper and electronic surveys to a total of 94 SET programs and 
received surveys from 65 programs, for a program response rate of 69 percent. This process 
yielded a total of 1,060 youth surveys, a sample large enough to allow for statistically sound 
comparisons of multiple subgroups on survey items (Exhibit ES1). 


Exhibit ES1 
Program and Youth Survey Response Counts 


Program 
Bealon Programs ine Youth survey 
g surveyed g responses 
surveys 


Exhibit reads: Nineteen programs in the Northeast region were 
surveyed and16 of these programs returned a total of 482 completed 
surveys. 


Representativeness of data. At this time, because 4-H does not have a uniform way of 
collecting data on individual participants, it was not possible for evaluators to select a nationally 
representative sample of youth for participation in the survey.” For this reason, the survey data 
presented in this report only speak to the population of participants surveyed and are not 
generalizable to the entire 4-H SET participant population. 


Statistical tests employed. Data in this report include descriptive statistics from the first 
administration of the YEAK survey. Evaluators also explored the relationships between various 
youth characteristics and youth responses to survey items using independent samples t-test or 
analysis of variance (ANOVA) for continuous variables and chi-square tests for categorical 
variables. Where statistically significant differences were found (using the threshold of p<0.05), 
we computed an effect size to measure the magnitude or strength of the finding. 


The statistical tests and measures of effect size used in analysis varied based on the 
properties of the data analyzed. For analyses of continuous variables, we computed a Cohen’s d 
measure of effect. For categorical variables we calculated a Cramer’s V effect. Conventions for 
educational research suggest that effect size values between 0.10 and 0.20 indicate a “small but 
meaningful” association, between 0.21 and 0.50 an “important” association, and 0.51 or higher 
an “impressive” association (Cohen, 1988; Lipsey, 1990). This report focuses on findings with 


"Details about the sample selection and survey administration can be found in Appendix A. 


an effect size of at least 0.20; comparisons or associations below this threshold were considered 
too weak to warrant reporting. 


In addition, evaluators used multiple regression to further explore the report’s findings by 
controlling for certain youth characteristics that could affect survey responses. This method 
produces an estimate of the impact of each element in the regression model, while all other 
elements in the model are held equal. In this report, an R’ statistic is displayed along with each 
regression result. This statistic represents the amount of variation in a particular outcome that 
the regression model can explain. For example, a regression model with an R’ statistic of 0.20 
concerning a certain survey question could explain twenty percent of the variation in youth’s 
responses to that question. This means that approximately eighty percent of the variation in 
youth’s responses is due to factors that were not in the model (and were not captured by the 
survey). We also report the beta (b) statistic for each regression finding, which explains the 
influence of a particular factor on youth’s survey responses. 


Analysis approach. We explored the associations between a variety of factors that might 
influence youth’s engagement, attitudes, aspirations, and knowledge regarding SET, including: 


: Respondents’ gender, age, and race/ethnicity. Responses from youth ages 9 to 
12 were compared with those from youth ages 13 to 18. In addition, youth 
responses were compared based on their self-reported race/ethnicity; survey 
responses from the two largest respondent groups, white youth and African 
American youth, were compared. 


= Respondents’ educational aspirations. Youth’s educational aspirations were 
measured by the survey question, “How far do you want to go in school?” The 
five response categories for this question were combined into three for analysis 
purposes: (1) graduate from high school, go to a trade or vocational school, or go 
to college for a little while; (2) finish college; and (3) get more education after 
college. 


a Respondents’ exposure to their SET program. In order to obtain a more complete 
picture of the amount of time respondents spent in their SET programs, we 
combined responses to two questions, “In general, how many hours do you spend 


> Beta, b, is a coefficient used in regressions to indicate the amount of change in an outcome that can be attributed to 
a predictor (independent) variable, when other predictors are held constant. 


In regressions with continuous outcome (dependent) variables, b can be interpreted in terms of the outcome 
variable. For example, if a predictor variable has a b of 2.5, for every | point increase in that predictor variable, the 
outcome variable will increase by 2.5 points. 


A transformed b coefficient, called exponential b (exp b) is used in regressions with categorical outcome variables 
and can be interpreted as the change in the odds of an outcome occurring. An exp b of greater than one indicates 
that as the predictor increases, the odds of the outcome occurring also increase. If exp b is less than one, as the 
predictor increases, the odds of the outcome occurring decrease (Field, 2005). 


Further details about regression results can be found in Appendix E. 


in this program/project each week?”, and “How long have you been participating 
in this science, engineering or technology program/ project?”, to create an index 
that rates their self-reported degree of exposure to SET programming. Youth who 
reported participating in their program for the greatest number of hours and 
months were categorized as “high exposure.” Youth who participated for fewer 
hours per week and for fewer months were categorized as “moderate exposure,” 
and youth who reported participating in their program the least were categorized 
as “low exposure.” Thirty percent of respondents were categorized as high 
exposure, 31 percent as moderate exposure, and 40 percent as low exposure.” 


: Types of 4-H programs youth are involved in. In the YEAK survey, youth were 
able to indicate the types of 4-H activities they have been involved in, including: 
clubs, camps, after-school programs, local fairs or events, community service 
projects, and working on their projects at home. Youth were compared based on 
whether or not they were involved in 4-H clubs and whether or not they were 
involved in 4-H after-school programs. 


: The type of community in which the program was located. Information about 
community type came from the Program Survey that was administered to county- 
and state-level 4-H Staff in fall 2009. For analysis purposes, the five original 
categories were combined into three: (1) farming community, town of under 
10,000 people or rural non-farm; (2) town, city, or suburb of a city with 10,000 to 
50,000 people; and (3) suburb or city with more than 50,000 people. 


There were some relationships between the factors listed above. If youth reported being 
involved in clubs, they were more likely to be in the “high exposure” group than youth who were 
not involved in clubs. Involvement in 4-H clubs was also associated with the type of community 
in which youth’s SET programs were located: youth involved in clubs were more likely to be in 
programs in middle-sized towns or cities than in large suburbs or cities. In addition, a greater 
proportion of white youth than African American youth have been involved in 4-H clubs, and a 
greater proportion of African American youth than white youth have been involved in 4-H after- 
school programs. Community type and race/ethnicity were also associated with one another. 
These relationships should be kept in mind when considering the findings in this report. 


Interpreting results. While the associations and regression models discussed in this 
report describe notable relationships between youth characteristics and survey responses, they 
should not in any instance be interpreted as implying causation. 


* Details about the exposure index can be found in Appendix C. 


Youth Characteristics: Key Findings 
Youth Demographics 


Youth who completed the survey participated in 4-H SET programs located in a variety 
of communities. About half of youth participated in programs that were located in farming 
communities or towns with less than 10,000 people (52 percent of youth). One-quarter of youth 
participated in programs that were located in towns, cities, or suburbs of cities with between 
10,000 and 50,000 people, and 23 percent of youth participated in programs that were located in 
suburbs or cities with more than 50,000 people. 


The SET participants surveyed included a slightly higher percentage of females than males 
(Exhibit ES2). More than half of the youth were between the ages of 9 and 12 (62 percent) and the 
remaining youth were between the ages of 13 and 18 (38 percent). The majority of SET 
participants identified themselves as white (68 percent), followed by African-American (19 
percent), and Latino/a (9 percent). Nearly 90 percent of SET participants reported being enrolled 
in a public school, with the next largest group indicating that they are home schooled (7 percent). 


Exhibit ES2 
Youth Demographic Characteristics 


Percent of 
Respondents 
(n=1,060) 
i 
Gender 


a a 
[snciame | 


Exhibit reads: Fifty-four percent of respondents were female. 


Exposure to 4-H and 4-H SET Programming 


While the majority of survey respondents had previously been enrolled in 4-H 
programming, there were many respondents enrolled in 4-H SET programs who reported that 
they were either new to 4-H or new to 4-H SET programming. Forty-one percent of youth said 
that they were currently enrolled in their first year of 4-H programming and 63 percent reported 
that they were currently enrolled in a 4-H SET program for the first time. 


When asked about the types of 4-H programming they are involved in, more than half of 
youth said that they are involved in 4-H clubs (55 percent), 42 percent reported being involved in 
after-school programs, and 32 percent said that they were involved in local fairs and events. 


Youth Attitudes: Key Findings 
Educational and Career Aspirations 


SET participants who responded to the survey reported having high educational 
aspirations. More than 80 percent of respondents indicated that they intend to finish college or 
continue to get more education after college. Nine percent of youth said that they would likely 
attend some college or attend a trade or vocational school, and another nine percent of youth 
indicated that they only intended to finish high school. 


When asked what type of career they anticipated pursuing in the future, respondents gave 
a wide range of answers, with similar numbers of youth expressing interest in a career in the 
medical field, the military, and the arts (items adapted from: Tai, 2006). When the career 
choices were collapsed into SET and non-SET careers, 50 percent of respondents reported 
wanting to pursue a SET career. 


Gender. Further analyses found that there were significant differences between the 
career aspirations of boys and girls: 61 percent of boys said they would like to pursue a SET 
career while only 40 percent girls said the same. Using regression analysis, respondents’ gender 
was found to be a significant predictor of their aspiration to a SET career. Controlling for other 
significant predictors, boys were three times more likely than girls to aspire to SET careers (exp 
b=3.10, R°=0.18).° 


° Significant predictors in this regression model were: gender, race, educational aspirations, exposure to SET 
programming, involvement in 4-H afterschool, and the type of community in which respondents’ programs were 
located. Respondents’ age and involvement in clubs did not have a significant impact on their aspirations toward 
SET careers. Details about this regression model can be found in Exhibit El of Appendix E. 


Attitudes Toward Science 


Youth were asked to respond to a set of items in order to gauge their interest in and 
enthusiasm for science. These items were taken from the National Assessment of Education 
Progress (NAEP) science exam, a subject-based assessment that is administered periodically to a 
nationally representative sample of fourth, eighth, and twelfth grade students. 


Seventy-one percent of SET participants said that science is one of their favorite subjects, 
68 percent said that they do science-related activities that are not for school work, and 59 percent 
said that they would like to have a job related to science when they graduate from school. 


Exposure to 4-H SET programming. Data analysis showed that the amount of time a 
respondent has spent in their SET program was related to their expressed interest in having “a 
job related to science.” Youth in the high exposure group were more likely to strongly agree or 
agree that they wanted to pursue a job in science than those youth who were in the low exposure 
group (73 percent versus 46 percent). Also, regression analyses showed that being in the low 
exposure group made respondents about half as likely to want a job related to science compared 
to other youth, when other significant factors were held constant (exp b=0.48, R°=0.13).° 


Comparison against a national sample. SET participants were separated by age into 
three groups and their responses were compared with those of 2005 NAEP respondents of similar 
ages (Exhibits ES3, ES4, and ES5).’ While the differences between the SET participants and 
NAEP respondents were statistically significant, these differences should be interpreted with 
caution because the effect sizes were not strong enough to meet this evaluation’s threshold.* 


° Significant predictors in this regression model were: exposure to SET programming, gender, race, educational 
aspirations, involvement in 4-H afterschool, and the type of community in which respondents’ programs were 
located. Respondents’ age and involvement in clubs did not have significant impacts on youth’s interest in having a 
job related to science. Details about this regression model can be found in Exhibit E2 of Appendix E. 

’ Data downloaded from the NAEP Data Explorer at http://nces.ed.gov/nationsreportcard/nde, on December 16, 
2009. 

* The number of SET respondents in each age group was not large enough to yield strong effect sizes when 
compared with the large group of NAEP respondents. 


Exhibit ES3 
Fourth Grade SET and NAEP Respondent Attitudes Toward Science 
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Fourth Grade SET Respondents (n=303) @ National Sample (n=147,700) 


Exhibit reads: Seventy-six percent of fourth grade 4-H respondents agreed with the statement “I like science” 
compared to 64 percent of youth in the national sample. 


Note: National sample data are from the NAEP science assessment, administered to students in grades 4, 8, 
and 12 in 2005. 4-H respondents ages 9 and 10 were classified as fourth graders for this analysis. 


Exhibit ES4 
Eighth Grade SET and NAEP Respondent Attitudes Toward Science 
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Exhibit reads: Seventy-four percent of eighth grade 4-H SET respondents agreed with the statement “I like 
science” compared to 50 percent of youth in the national sample. 


Note: National sample data are from the NAEP science assessment, administered to youth in grades 4, 8, 
and 12 in 2005. 4-H respondents ages 13 and 14 were classified as eighth graders for this analysis. 
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Exhibit ES5 
Twelfth Grade SET and NAEP Respondent Attitudes Toward Science 
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Exhibit reads: Eighty-six percent of twelfth grade 4-H SET respondents agreed with the statement “I like science” 
compared to 31 percent in the national sample. 


Note: National sample data are from the NAEP science assessment, administered to students in grades 4, 8, and 12 
in 2005. 4-H respondents ages 17 and 18 were classified as twelfth graders for this analysis. 


Exposure to Informal SET Activities: Key Findings 


In order to assess the extent to which SET participants have been exposed to informal 
learning opportunities related to SET content, the YEAK survey asked participants to report their 
involvement in SET-related activities. 


Respondents were given a list of SET-related activities and asked to indicate whether they 
had previously participated in any of the activities. A variety of activities were included, from 
visiting an aquarium or zoo to taking apart a device in order to determine how it works. Overall, 
youth reported having been involved in a large number of informal SET activities, with the average 
respondent reporting that they had participated in 9 of the 13 listed activities. 


One of the long-term goals identified in the 4-H SET Logic Model is that youth 
participating in SET programs should become increasingly aware “of opportunities to contribute 
to society using SET skills.” Survey respondents were asked whether or not they participated in 
a series of science leadership and community activities (items adapted from Silliman, 2010). 
Many SET participants reported being involved in such activities: more than three-quarters of 
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respondents reported that they had helped with a community service project related to science 
and more than half said that they had taught others about science (Exhibit ES6). 


Exhibit ES6 
Participation in Science Leadership Activities 


Percent of 
Respondents 
(n=1,010) 


Helped with a community service project related to science 


Taught others about science 
Used science tools to help the community 
Organized or led science-related events 


Exhibit reads: Seventy-six percent of respondents have helped with a community 
service project related to science in the past year. 


Race and ethnicity. When compared to their white peers, African American respondents 
reported participating in higher average numbers of science leadership activities. African 
American participants reported that they were involved in an average of 2.5 science leadership 
activities while white participants said they were involved in an average of 2.0 activities. 


Educational aspirations. There were differences between youth’s participation in 
informal SET and science leadership activities based on their educational aspirations. The more 
education youth reported wanting to pursue, the more likely they were to participate in informal 
SET activities and science leadership activities. Youth who want to get more education after 
college reported participating in an average of 10 informal SET activities, while youth who do 
not expect to earn college degrees participated in an average of 8 activities. Similarly, youth 
who want to get more education after college said that they participated in an average of 2.2 
science leadership activities, while their peers who do not expect to earn college degrees 
participated in an average of 1.8 activities. In addition, regression analysis showed that wanting 
to get more education after college was associated with participating in 1.2 more informal SET 
activities, when other significant predictors were held constant (b=1.2, R°=0.17).” 


* Significant predictors in this regression model were: educational aspirations, level of exposure to SET programming, 
race, involvement in afterschool, age, and the type of community in which respondents’ programs were located. 
Respondents’ gender and involvement in clubs did not have significant impacts on youth’s participation in informal 
SET activities. Details about this regression model can be found in Exhibit E5 of Appendix E. 


12 


Exposure to 4-H SET programming. Youth with more exposure to 4-H SET 
programming participated in more informal SET and science leadership activities than youth 
with less exposure to SET programming. Youth in the high exposure group participated in an 
average of 10 informal SET activities and 2.3 science leadership activities, while their peers in 
the low exposure group participated in an average of 9 informal SET activities and 1.8 science 
leadership activities. Regression analysis supported this finding: when other significant 
predictors were held constant, being in the high exposure group was associated with participating 
in 1.0 more informal SET activities (b=1.0, R°=0.17)."° 


Types of 4-H program involvement. On average, youth who reported being involved in 
4-H clubs participated in more informal SET activities than did youth who were not involved in 
4-H clubs. Youth involved in clubs participated in an average of 10 informal SET activities, 
compared to an average of 9 activities for youth who were not involved in clubs. In addition, 
youth involved in clubs participated in 2.1 science leadership activities on average, compared to 
an average of 1.9 activities for youth not involved in clubs. However, regression analysis did not 
show involvement in clubs to significantly predict participation in informal SET activities. 


Youth involved in after-school 4-H SET programs participated in an average of 9 
informal SET activities, compared to an average of 10 activities for youth who were not involved 
in after-school programs. Regression analysis also showed that being involved in after-school 
was associated with participating in 0.8 fewer informal SET activities, controlling for other 
significant factors (b=-0.8, R7=0.17).!! 


Community type. Youth whose programs were located in farming communities or towns 
of less than 10,000 people participated in more informal SET activities on average (10) than did 
youth whose programs were located in suburbs or cities of more than 50,000 people (8). Youth 
whose programs are located in towns or cities with between 10,000 and 50,000 people also 
reported taking part in more informal SET activities (10) than youth in programs located in large 
cities (8). When other significant predictors were held constant, attending a program located in a 
large city was associated with participating in 1.8 fewer informal SET activities (b=-1.8, 
R°=0.17). 


Youth whose programs are located in farming communities or small towns also reported 
participating in more science leadership activities than youth in programs located in large cities 
(2.2 science leadership activities compared to 1.7 activities, respectively). 


'° Details about this regression model can be found in Exhibit E5 of Appendix E. 
'' Details about this regression model can be found in Exhibit ES of Appendix E. 
'? Details about this regression model can be found in Exhibit ES of Appendix E. 
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Skills and Knowledge: Key Findings 


4-H is committed to developing life skills in young people and empowering them to 
make sound decisions and thoughtful choices. The SET logic model articulates the 
development and application of life skills as key outcomes for youth participating in SET 
programs. Life skills such as decision making, problem solving, and critical thinking are 
aligned with the learning opportunities 4-H expects SET programs to provide. 


Life Skills 


A majority of respondents reported having many of the life skills that 4-H programming 
aspires to instill in youth. More than three-quarters of youth reported that: when they have a 
decision to make, they always think before making a choice; they always keep their mind open 
to different ideas when planning to make a decision; they first figure out exactly what the 
problem is when trying to solve a problem; and they try to determine what caused a problem 
when they try to solve it. 


SET Skills and Abilities 


The survey asked a subset of questions of youth ages 9 to 12 to assess their ability to 
perform certain science process skills. Youth were most likely to report that they are able to 
make a chart or picture to show information (88 percent). More than three-quarters of youth 
reported that they can: write down information correctly, do an experiment to answer a 
question, tell others how to do an experiment, and explain why things happen in an experiment. 


SET participants ages 13 to 18 were asked a similar set of questions about their mastery 
of certain science process skills (items adapted from: Arnold & Bordeau, 2009). Most of these 
older youth reported that they were always or usually capable of: recording data accurately (76 
percent), using data to create a graph for presentation to others (75 percent), and using the 
results of an investigation to answer the questions asked (73 percent) (Exhibit ES7). 
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Exhibit ES7 
Science Process Inventory, Ages 13-18 


Percent of Respondents, Ages 13-18 
(n=382) 


Always/Usually 
| can record data accurately 


| can use data to create a graph for presentation to others 


| can use the results of my investigation to answer the questions | asked 


| can create a display to communicate my data and observations 
| can analyze the results of a scientific investigation 

I can use models to explain my results 

| can ask a question that can be answered by collecting data 

| can use scientific knowledge to form a question 

| can use science terms to share my results 

| can design a scientific procedure to answer a question 


| can communicate a scientific procedure to others 


Exhibit reads: Seventy-six percent of respondents ages 13-18 report that they can always or usually record data 
accurately. 


In order to analyze responses to the science-process skills inventory, we computed two 
scales representing youth responses. For respondents ages 9-12, the scale ranges from zero to 
five where a score of zero indicates that the respondent did not report being able to do any of the 
listed science-process tasks and a score of five indicates that the respondent reported being able 
to do all of the science-process tasks. For respondents ages 13-18, a mean scale was created 
where youth responses were averaged across all 11 items; a response of “always” received a four 
and “never” received a one.’? 


Educational aspirations. Across both age groups, youth who said they planned to pursue 
education beyond college rated their science-process skills more highly than those youth who 
said that they did not plan to complete college. Respondents ages 9-12 who want to get more 
education after college said “yes” to an average of 4.4 items out of 5, while youth who do not 
plan to complete college said “yes” to an average of 3.8 items. Youth between the ages of 13 
and 18 who want to get more education after college scored an average of 3.2 out of 4 on the 
science-process scale, while youth who do not plan to complete college averaged a score of 2.4. 


'S Details about scales can be found in Appendix B. 
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These findings were supported in regression analyses. For younger youth, wanting to get 
more education after college was associated with a 0.7 point increase out of 5 points, when 
controlling for other significant factors (b=0.69, R’=0.11).'* Also, wanting to finish college was 
associated with an increase of 0.4 points out of five for younger youth (b=0.35). Controlling for 
other significant factors, wanting to get more education after college was associated with an 
increase of 0.3 points out of 4 on the SET-related skills scale (b=0.29, R7=0.16).'° Also, not 
aspiring to finish college was associated with a decrease of 0.4 points (b=0.-43). 


Community type. Youth ages 9-12 from programs located in farming communities or 
towns of less than 10,000 people rated their SET-related skills more highly than did youth whose 
programs were located in suburbs or cities of more than 50,000 people (4.3 out of 5, compared to 
3.4). Regression analysis found that, controlling other significant factors, participating in a 
program located in a suburb or large city was associated with a decrease of 1.0 points out of 5 on 
younger youth’s ratings of their SET skills (b=-0.95, R’=0.11)."° 


SET Program Environment: Key Findings 


In order for out-of-school time and enrichment programs such as 4-H to be most 
effective, they need to create a healthy, positive program environment by employing practices 
that have been shown to support youth development. Like all 4-H programs, 4-H SET programs 
are designed to focus on youth development, on experiential learning, and on creating supportive 
environments for youth. 


When asked to choose the three things they liked best about their SET program, 62 
percent of respondents cited the opportunity to spend time with their friends (Exhibit ES8). A 
similar number reported that the opportunity to do hands-on activities and projects ranked among 
their favorite aspects of their SET program (60 percent). For 30 percent of youth, the presence 
of kind, caring adults was a favorite aspect of their program. In addition, 30 percent of 
participants said a favorite part of their program was that it is “a group where I feel like I 
belong.” 


'* Significant predictors in this regression model were: educational aspirations, youth’s exposure to SET programs, 
and the type of community in which their program was located. Respondents’ gender, race, and involvement in 
clubs or afterschool did not have significant impacts on younger youth’s assessment of their SET skills. Details 
about this regression model can be found in Exhibit E6 of Appendix E. 

'S Significant predictors in this regression model were: exposure to SET programs and educational aspirations. 
Respondents’ gender, race, involvement in clubs or afterschool, and the type of community in which their SET 
program was located did not have significant impacts on older youth’s score on the SET-related skills scale. Details 
about this regression model can be found in Exhibit E7 of Appendix E. 

'° Details about this regression model can be found in Exhibit E6 of Appendix E. 
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Exhibit ES8 
Favorite Characteristics of this SET Program 


Respondents (n=965) 
| get to spend time with my friends 
| get to do hands-on activities and projects 
The adults are caring and kind 
It is a group where | feel like | belong 


| can use tools and materials here that | don’t have at school 28 

or home 

| get opportunities to demonstrate things | have learned or 26 

made in front of others 

It is a place where | feel safe 
| get positive feedback from adults and other kids 
| get to do community service 
| like the curriculum/project book a ae 


Exhibit reads: Sixty-two percent of respondents said that one thing they like best about 
coming to their 4-H SET program is that they get to spend time with their friends. 


Note: Because youth could select up to three responses, percents do not add up to 100. 


The survey asked youth to rate their experiences in 4-H SET on a series of items 
measuring the environment of their program (items adapted from: Silliman, 2008). Overall, 
participants surveyed gave positive reports of their program environments: 69 percent of youth 
said that in their program, all kinds of kids are always welcome (Exhibit ES9). More than half of 
youth said that the following are “always” true in their programs: that it is OK to make mistakes, 
that they are encouraged to take responsibility, and that they feel safe and respected. 
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Exhibit ES9 
Youth Assessments of Program Environment 


All kinds of kids are welcome 


Itis OKto make mistakes 


lam encouraged to take 
responsibility 


| feel safe and respected 


Adults listen to what | have to say 


| feel comfortable going to adults 
for advice 


| feel like |can make adifference 


Other kids care about me 


lam afraid | will be embarassed 
orputdown 


20 40 60 80 


Percent of Respondents 


@ Always B Usually 


Exhibit reads: Sixty-nine percent of respondents reported that all kinds of kids are always welcome. 


As stated above, participants value the opportunity their SET program gives them to 
spend time with friends. Almost all youth (94 percent) reported that that in their 4-H SET 
program, they can learn with their friends (Exhibit ES10). In addition, the vast majority of youth 
said that at their program they: can see science in a fun way, get answers to their questions from 
leaders, and do hands-on science activities. 
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Exhibit ES10 
Opportunities in SET Programs 


In this 4-H program or project, | can... 


Learn with my friends 


See scienceina fun way 


Get answers to my questions from 
leaders 


Do hands-on science activities 


Tell agroup of people about 
something | learned ormade 


Do experiments 
Solve problems 
Serve my community 


Learn about careers 


Percent of Respondents 


Strongly Agree B Agree 


Exhibit reads: Forty-eight percent of respondents “strongly agreed” that in their SET program, they are able to learn 
with their friends. 


In order to create an overall measure of participant evaluations of their program 
environment, the items shown in Exhibit ES9 above were combined into a scale in which an 
“always” response received four points and “never” received one point (the question “I am afraid 
I will be embarrassed or put down” was reverse-coded). When participant responses were 
combined into this scale, the average score was 3.3, indicating that respondents on the whole feel 
positively about their program. This scale was then used to compare how different groups of 
respondents rated their programs. 


Educational aspirations. Participant ratings of their program environment differed by 
their educational aspirations. The further youth said they wanted to go in school, the better, on 
average, they rated their program environment. Youth who said they wanted to get more 
education after college had an average scale score of 3.3 out of 4 on the program environment 
scale, compared to an average of 3.1 for youth who reported they did not aspire to finish college. 
Youth who said they wanted to finish college also scored higher on average (3.3 out of 4) than 
youth who did not. Regression analysis showed that, controlling for other significant factors, not 
aspiring to finish college was associated with a decrease of 0.3 points on the program 
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environment scale (b=-0.31, R°=0.11)."” Aspiring to finish college (but not get more education 
after college) was associated with a decrease of 0.1 points on the scale (b=-0.13). 


Exposure to 4-H SET programming. On average, the more time participants reported 
spending in their SET program, the better they rated their program’s environment. Youth in the 
high exposure group scored an average of 3.4 out of 4 points on the program environment scale, 
which was better than the average rating the low exposure group gave their programs (3.2 out of 
4). Youth in the moderate exposure group also tended to rate their program environment higher 
than did those youth in the low exposure group (3.3 out of 4). Regression analysis showed that, 
controlling for other significant factors, being in the high exposure group was associated with an 
increase of 0.1 points on the program environment scale (b=0.10, R°=0.11).'® 


Community type. Youth whose programs were located in medium-sized towns or cities 
rated their program environment highest on average, compared to youth from programs located 
in large suburbs or cities, and to youth whose programs were located in farming communities or 
small towns. Youth in medium-sized towns or cities averaged 3.4 points out of 4 on the program 
environment scale, compared to 3.3 for youth in farming communities or small towns, and to 3.1 
for youth in large suburbs or cities. Youth in farming communities or small towns also rated 
their program environments higher than did youth in large cities (3.3 compared to 3.1 out of 4). 
Regression analysis showed that participating in a program in a medium-sized town was 
associated with an increase of 0.1 points on the program environment scale (controlling for other 
significant factors) (b=0.09, R’=0.1 ibe 


Conclusions 


Most 4-H SET participants surveyed for this evaluation are enthusiastic about SET, have 
SET skills, and many are interested in pursuing SET careers. Evaluators found potentially 
important relationships between youth characteristics and survey responses. However, these 
results should be interpreted with caution as they do not imply causation. 


7 Youth who completed the survey participated in 4-H SET programs located in a 
variety of communities, from farming communities and small towns to large 
urban centers. Youth surveyed included a slightly higher percentage of females 
than males, and the majority of respondents identified themselves as white (68 
percent), followed by African-American (19 percent), and Latino/a (9 percent). 


7 SET participants who responded to the survey reported having high educational 
aspirations. More than 80 percent of respondents indicated that they intend to 
finish college or continue to get more education after college. 


' Significant predictors in this regression model were: educational aspirations, involvement in clubs, exposure to 
SET programming, race, and community type. Respondents’ gender and involvement in afterschool did not have 
significant impacts on youth’s ratings of their program environment. Details about this regression model can be 
found in Exhibit E8 of Appendix E. 

'’ Details about this regression model can be found in Exhibit E8 of Appendix E. 

'° Details about this regression model can be found in Exhibit E8 of Appendix E. 
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The majority of respondents said that science was one of their favorite subjects, 
that they do science-related activities that are not for school work, and that they 
would like to have a job related to science. 


Overall, youth reported having been involved in a large number of informal SET 
activities in their lives. The majority of youth also reported that they had helped 
with a community service project related to science and more than half said that 

they had taught others about science. 


Participants rated themselves highly on a variety of life skills and also gave 
themselves high marks on an inventory of science process skills. 


Participants gave positive reports of their program environments. Their favorite 
aspects of their programs are the relationships they form with adults and youth 
and the opportunities they have to do hands-on science activities. 


Demographic Characteristics 


Female participants were less likely than their male counterparts to anticipate 
pursuing a SET-related career. 


African American respondents were more likely than their white peers to report 
that they take science courses only to meet requirements. 


When compared to their white peers, African American respondents reported 
participating in higher average numbers of science leadership activities. 


Educational Aspirations 


The more education youth reported wanting to pursue, the more likely they were 
to participate in informal SET activities and science leadership activities. 


The more education youth reported wanting to pursue, the higher youth rated their 
science process skills. 


The more education youth reported wanting to pursue, the better, on average, they 
rated their program environment. 


Exposure to SET programming 


Youth with higher levels of exposure to their SET programs were more likely to 
report wanting to pursue a job in science than were youth with less exposure. 
Youth whose interest in SET has been sparked may be more likely to spend time 
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in a SET program, or SET programs may be affecting participants’ career 
aspirations. 


Youth with higher levels of exposure to their SET programs participated, on 
average, in a greater number of informal SET activities than youth with less 
exposure to their programs. 


Youth with higher levels of exposure to their SET programs rated their life skills 
higher than did youth in the low exposure group. 


The more time youth reported spending in their SET program, the higher they 
rated their program’s environment. 


Involvement in 4-H Clubs 


Youth involved in clubs rated their life skills higher than did youth who were not 
involved in clubs. 


Youth involved in clubs reported being involved in more informal SET activities 
than youth who were not involved in clubs. It could be true that youth involved in 
clubs are already interested in pursuing science learning opportunities before they 
enroll in their 4-H SET program, or it could be that involvement in clubs inspires 
youth to explore science, engineering, and technology in their lives outside of 4- 
H. 


Community Type 


Youth in programs located in farming communities, small towns, and medium- 
sized cities participated in more informal SET activities than youth in programs 
located in suburbs or large cities. 


Youth in programs located in farming communities or small towns rated their 
science process skills more highly than other youth. 


Youth in programs located in medium-sized towns or cities rated their program 


environment highest on average, compared to youth from programs in large cities 
and farming communities or small towns. 
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Recommendations 


7 The difference in SET career aspirations found in this study based on gender is in 
agreement with other research showing that eighth-grade girls were less likely to 
indicate interest in a mathematics- or science-related career than were boys with 
similar levels of achievement, attitudes toward their mathematics class, and 
family background (Catsambis, 1994). 4-H SET programs are uniquely poised to 
encourage girls who have expressed interest in SET fields by enrolling in SET 
programs to further pursue education and career opportunities. 


7 The differences between white and African American respondents’ reasons for 
pursuing science courses reflect other research showing that students of color are 
still less likely to take advanced science courses than their white counterparts. 
Data from the National Center for Education Statistics (NCES) show that the 
number of high school youth completing higher level mathematics and science 
courses has increased over the past 20 years; however, there are continuing 
disparities by race, especially at the highest course levels. A 2007 analysis of 
NCES data revealed that there were significantly fewer African American and 
Latino/a students taking advanced mathematics and science classes than white and 
Asian American students. 


Because 4-H SET programs offer informal, out-of-school opportunities for youth 
to engage in SET content, 4-H is well positioned to help generate genuine interest 
and excitement among African American youth that can help shape their interests 
in pursuing SET education and careers. 


7 Research suggests that career aspirations are often solidified in early adolescence 
(Kerka, 1994). In order to ensure that 4-H SET participants consider pursuing 
careers in the SET fields, SET programming should include ample opportunities 
for youth to not only learn about careers in the SET fields, but also about the 
educational trajectories that they must follow in order to successfully pursue these 
careers. 


7 By providing engaging out-of-school programming, 4-H SET programs have the 
potential to bolster participants’ interest in pursuing education and careers in the 
SET fields. Continuing to encourage 4-H SET programs to employ best practices 
will help ensure that programs inspire life-long interest in SET among all 
participants. 


7 For many of the youth in this study, the opportunity to do hands-on learning 
activities formed a major part of their attraction to 4-H. 4-H has the potential to 
positively impact youth’s interest in SET by giving them the opportunity to learn 
about the SET fields in a hands-on manner. SET programs, regardless of delivery 
method, should be set up to provide active learning opportunities for youth. 4-H 
may want to consider how youth-centered SET programming could best be 
implemented using various delivery methods. 
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In addition to the opportunity to participate in hands-on learning activities, SET 
participants reported that the relationships they have with other youth and with 
adults in their programs are important to them. By fostering healthy, supportive 
relationships among youth and between youth and adults, programs lay the 
groundwork for engaging youth in SET. 


Since the types of communities in which respondents’ SET programs were 
located were associated with different youth responses to survey measures, 4-H 
may want to examine the types of programming being offered in different 
communities to ensure that 4-H’s flagship delivery methods are available to all 
participants. 


Data on youth participation in SET programs, as well as data on program 


implementation, would enhance the power of the evaluation and allow evaluators 


to tie youth outcomes directly to participation in SET programs. 
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